Analyzing conversations to automatically identify action items

ABSTRACT

An action item identification system automatically determines action items by analyzing conversations of representatives with customers. The action item identification system retrieves recordings of various conversations, extracts features of each of the conversations, and analyzes the features to determine a set of features that is indicative of an action item associated with the corresponding conversation. The set of features is further analyzed to generate the action item in an action item manifest (a) as a summary of what is discussed in the conversations or (b) verbatim from the conversations.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation-in-part of U.S. application Ser. No.15/168,675 titled “MODELING VOICE CALLS TO IMPROVE AN OUTCOME OF A CALLBETWEEN A SALES REPRESENTATIVE AND A CUSTOMER” filed May 31, 2016, whichclaims the benefit of U.S. Provisional Application Ser. No. 62/169,456titled “MODELING VOICE CALLS TO IMPROVE AN OUTCOME OF A CALL BETWEEN ASALES REPRESENTATIVE AND A CUSTOMER” filed Jun. 1, 2015, and U.S.Provisional Application Ser. No. 62/169,445 titled “COORDINATING VOICECALLS BETWEEN SALES REPRESENTATIVES AND CUSTOMERS TO INFLUENCE ANOUTCOME OF THE CALL” filed Jun. 1, 2015, all of which are incorporatedherein by reference for all purposes in their entirety.

BACKGROUND

With over 2.4 million non-retail inside sales representatives in theUnited States (U.S.) alone, millions of sales phone conversations aremade on a daily basis.^(i) However, except for rudimentary statisticsconcerning e.g., call length and spotted keywords and phrases, salesconversations are left largely unanalyzed, rendering their contentinaccessible to modeling, and precluding the ability to optimize themfor desired outcomes. Recent advances in automatic speech recognition(ASR) technologies, and specifically in large vocabulary continuousspeech recognition (LVCSR), are for the first time enablinghigh-accuracy automatic transcription of conversations. At the sametime, natural language processing (NLP) approaches to both topicmodeling and world-knowledge modeling, have become much more efficientdue to the availability of large, freely accessible natural languagecorpora (e.g., CommonCrawl), as well as freely available ontologies or“knowledge graphs” (e.g., DBpedia). Finally, recent research on affectidentification applying machine learning (ML) has been able tosuccessfully model subjective aspects of emotion and personality traitsas perceived by listeners. ^(i)Insidesales.com “Market size 2013” study

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a call-modeling system in which thedisclosed embodiments can be implemented.

FIG. 2 is a block diagram of a feature generation component of thecall-modeling system for extracting features from call data, consistentwith various embodiments.

FIG. 3 is a block diagram of a classifier component of the call-modelingsystem for generating classifiers, consistent with various embodiments.

FIG. 4 is a block diagram of a real-time analysis component of thecall-modeling system for generating on-call guidance for arepresentative during a call between the representative and a customer,consistent with various embodiments.

FIG. 5 is a flow diagram of a process for performing offline analysis ofconversations between participants, consistent with various embodiments.

FIG. 6 is a flow diagram of a process for modeling calls between theparticipants to generate on-call guidance, consistent with variousembodiments.

FIG. 7 is a block diagram of an action item identification system,consistent with various embodiments.

FIG. 8 illustrates various examples of action items identified by theaction item identification system, consistent with various embodiments.

FIG. 9 is an example of action item data having features that areindicative of an action item, consistent with various embodiments.

FIG. 10 is a flow diagram of a process for determining action items byanalyzing conversations of representatives, consistent with variousembodiments.

FIG. 11 is a block diagram of a processing system that can implementoperations of the disclosed embodiments.

DETAILED DESCRIPTION

Embodiments are disclosed for a call-modeling system for modelingconversations, e.g., voice conversations, in real time, with the goal ofhelping users, e.g., sales representatives and/or their managers, toimprove and/or guide the outcome of conversations with other users,e.g., customers. One such embodiment can model the calls based oncharacteristics of the conversation, e.g., voice of the representativesand/or the customers, and content of the conversation, with the goal ofpositively influencing the outcome of the call. The call-modeling systemcan generate real-time probabilities for possible outcomes of areal-time conversation, e.g., an ongoing conversation between a specificrepresentative and a customer, and generate specific on-call guidance,which may be either conducive or detrimental to a desired conversationoutcome. The generated probabilities and on-call guidance may be used bythe representatives and/or their managers to either increase theprobability of a desired outcome and/or optimize the conversation for aspecified duration if the predicted outcome is not going to be a desiredoutcome. For example, for renewing a magazine subscription, thecall-modeling system can generate an on-call guidance suggesting arepresentative to engage in a rapport building exercise with thecustomer if it is determined that doing so increases the chances of thecustomer renewing the membership by 45%. On the other hand, if thecall-modeling system predicts from the on-going conversation that thecustomer is not going to renew the subscription membership, then thecall-modeling system can suggest the representative to wrap up theconversation as soon as possible.

The call-modeling system can include (i) an offline analysis componentand (ii) a real-time analysis component. The offline analysis componentcan take as input conversations between a calling party, e.g., acustomer, and a called party, e.g., a representative, and process theconversations using multiple distinct components to generate multiplefeatures of the conversations. In some embodiments, the conversationscan be audio recordings of calls between called party and the callingparty (collectively referred to as “participants”). The features caninclude transcripts of audio recordings, vocabulary, semanticinformation of conversations, summarizations of utterances and variousnatural language entailments, summarization of a call, voice signalassociated features (e.g., a speech rate, a speech volume, a tone, and atimber), emotions (e.g., fear, anger, happiness, timidity, fatigue),personality traits (e.g., trustworthiness, engagement, likeability,dominance, etc.), personal attributes (e.g., an age, an accent, and agender), customer-representative pair attributes that indicate specificattributes associated with both the speakers that contribute to aspecified outcome (e.g., similarity of speech rate between therepresentative and the customer, extrovert/introvert matching, or genderor age agreement).

Note that a recording of the conversation can be of a conversation thatis any of telephone based, Voice over Internet Protocol (VoIP) based,video conference based, Virtual Reality (VR) based, Augmented Reality(AR) based, or based on any online meetings, collaborations orinteractions, electronic mail (e-mail). The recording can also be of aconversation that has happened between two or more speakers physicallylocated in the same room. In some embodiments, a recording based on anyonline meetings, collaborations or interactions, or email can be atranscript of the corresponding interaction.

Further, the features can include not only aural features, but alsonon-aural features, e.g., visual features such as body language of aparticipant, and facial expressions of the participant, or anycombination of aural and non-aural features. The features could also begenerated from the transcripts of any of emails, online messages, andonline meetings. In some embodiments, any of a word, a phrase, a text,emoji, symbols, or a combination thereof can be used to determine aparticular feature. For example, it can be determined that a text suchas “Ha Ha” or “rofl” in the transcript can indicate laughter.

In some embodiments, the audio recordings can be tagged with varioustags, e.g., a tag that indicates a trait (e.g., “extrovert”,“trustworthy voice”, “anxious”, etc.) of one or more of theparticipants, a tag that indicates a call outcome (e.g., “sales closed”,“sales failed”, or “follow-up call scheduled”), and/or a tag thatindicates “key moments” of a conversation. A “key moment” or a “moment”can be a specific event or a specific characteristic which occurs in thecall. The event can be any event that is of specific interest for aspecific application for which the call-modeling system is beingimplemented. An administrator of the call-modeling system can configurewhat events in a call have to be identified as a moment. For example, amoment can be laughter, engagement, fast-talking, open-ended questions,objections, or any combination thereof in a conversation. In someembodiments, the moments are identified automatically by a momentidentification system. The tags can be generated automatically by thecall-modeling system, manually, e.g., by human judgment, or both. Insome embodiments, the tags are generated automatically by thecall-modeling system. The tag can include various details, e.g.,information regarding a moment, a time interval at which the momentoccurred in the call, duration for which the moment lasted, informationregarding the participants of the call, etc.

The moments can also be notified to and/or shared between theparticipants during an on-going conversation and/or after theconversation. For example, during a call between a customer and arepresentative, the call-modeling system can analyze the call, identifythe moments in the conversation, and tag, notify and/or share themoments with the representative's manager, peers or other people. Theshared moments can be used for various purposes, e.g., for coaching therepresentatives in handling the calls to improve outcomes of the callsbased on various situations. The moments can be shared using variousmeans, e.g., via email, a chat application, or a file sharingapplication.

In some embodiments, the offline analysis component uses distinctcomponents to extract the features. The components can include anautomatic speech recognition (ASR) component, which can produce atranscription of the conversation, a natural language processing (NLP)component, which can extract semantic information (such as open-endedquestions asked, key objections, etc.) from the conversation, an affectcomponent, which can analyze the recording for emotional signals andpersonality traits (e.g., likeability and trustworthiness), and ametadata component, which can extract data regarding conversation flow(i.e., who spoke when, and how much silence and overlap occurred).

The offline analysis component can analyze the features to generate oneor more classifiers that indicate conversation outcomes, e.g., “salesclosed”, “sales failed.” Each of the classifiers indicates a specificoutcome and can include a set of features that contribute to thespecific outcome. The offline analysis component can generate multipleclassifiers for the same outcome; however, the multiple classifiers canhave distinct sets of features. In some embodiments, the offlineanalysis component can analyze the features using a machine learningalgorithm (e.g., a linear classifier, such as a support vector machine(SVM), or a non-linear algorithm, such as a deep neural network (DNN) orone of its variants) to generate the classifiers.

In some embodiments, the offline analysis component generates aclassifier for different time intervals or time windows of theconversations. For example, the offline analysis component can analyzethe extracted features for seconds 00:05-00:10 of a conversation,seconds 00:20-00:30, and minutes 1:00-2:00, and generate a classifierfor each of those time windows. The offline analysis component feeds theextracted features into the machine-learning algorithm to producemultiple classifiers corresponding to the time windows. The time windowscan be of varying lengths or fixed lengths. In some embodiments, theoffline analysis component can generate classifiers for other relativepositions of a conversation. For example, the offline analysis componentcan generate a classifier corresponding to an instance in theconversation, e.g., when a customer spoke for the first time in theconversation, and analyze features such as a pitch of the voice, a topicthe customer spoke about first, and the length of the customer's firsttalk, to generate the classifier.

The real-time analysis component uses the classifiers to model areal-time conversation, e.g., an ongoing call between a representativeand a customer, that helps the representative to increase a probabilityof a desired outcome of the conversation or optimize the conversationduration in case the real-time analysis component does not predict thedesired outcome. The real time analysis component receives real-timecall data of an ongoing conversation between the customer and arepresentative and analyzes the real-time call data to generate a set offeatures, e.g., using the offline analysis component as described above.The real-time analysis component can then feed the features to theclassifiers to generate probabilities of potential outcomes of the call.The real-time analysis component can use the classifiers with highestprediction powers to generate the probabilities of various potentialoutcomes. In some embodiments, the real-time analysis component measuresthe prediction powers of the classifiers using an F-score, which, instatistical analysis, is a (possibly weighted) harmonic mean ofprecision and recall.

The real-time analysis component feeds the extracted features into theclassifiers with high F-scores to generate probabilities of possibleoutcomes. Based on the probabilities, the real-time analysis componentcan also generate on-call guidance, which encourages the representativeand/or their managers to modify, desist or persist with a specifiedon-call behavior to increase or decrease the probability of one of thepossible outcomes, e.g., a desired outcome such as closing a sale. Insome embodiments, the on-call guidance includes a set of suggestedfeatures and their values to be adopted, desisted or persisted with bythe representative. For example, the on-call guidance can includeinstructions for the representative to change the rate of speech (e.g.,speak slower), use specific key words, or pose more open-ended questionsto the customer.

In some embodiments, the on-call guidance can change as the callprogresses, e.g., based on the classifiers that are relevant to the callat that particular time of the conversation. For example, during thefirst two minutes of the call, a classifier that corresponds to thefirst two minutes of the call may be used to generate the on-callguidance such as instructing the representative to pose open-endedquestions to the customer, and then in the third minute, a classifierthat corresponds to the third minute of the call may be used to revisethe on-call guidance, e.g., suggest to the representative to adjust thespeech rate to match with that of the customer.

Additionally, if according to the classifiers, the real-time analysiscomponent predicts the conversation to fail, the on-call guidance maysuggest to the representative to quickly wrap up the call in order tospare the representative's time. The on-call guidance of the real-timeanalysis module may be presented on-screen or via any other interface(e.g., voice instructions given through an ear piece) to therepresentative and/or the manager. The embodiments can produce real-timeprobabilities of various outcomes of the conversations, enabling livecoaching that can help the representatives in improving the outcomes ofthe conversations in real-time.

Embodiments are also disclosed for an action item identification systemthat identifies action items based on an analysis of the conversationsbetween the representatives and the customers. In some embodiments, anaction item is an activity or a task to be performed by an entity, e.g.,a representative and/or a customer associated with a conversation. Forexample, an action item for a representative associated with aconversation can be “send an email to the customer regarding theestimate.” The action item identification system retrieves recordings ofvarious conversations, extracts features of each of the conversations,and analyzes the features to determine if any of the conversationsincludes features that are indicative of an action item resulting fromthe corresponding conversation. Upon identifying features in aconversation that are indicative of an action item, the action itemidentification system generates the action item based on the set offeatures, e.g., as an action item manifest.

In some embodiments, the action item identification system can alsoidentify a “next step,” based on the analysis of the conversations,which is indicative of what an entity should do subsequent to theconversation. The next step can include an action item. However, in someembodiments, the next step is not an action item. For example, a nextstep for a customer associated with a conversation can be “Wait toreceive an email regarding the estimate from the representative.” A nextstep that is not an action item may not require the entity to performany specific task. In some embodiments, a next step may be identifiedfor an entity even if no action item is identified for the entity but anaction item has been identified for another entity in the conversation.For example, if the action item identification system identifies anaction item such as “send an email to the prospect” is identified for arepresentative in the conversation, then the action item identificationsystem can identify a next step for the customer in the conversation as“wait for the email from the representative,” regardless of whether anaction item is identified for the customer or not.

The action item identification system can send the action item manifest,which includes information regarding one or more action items identifiedin a conversation, to a consumer user, who can use the action itemmanifest for various purposes. A consumer user can be a representative,a sales manager, or a vice president of sales, or any other user whoconsumes information regarding the action item for a specifiedconversation. The action item identification system can notify theconsumer user regarding the action item manifest in various ways. Forexample, the action item identification system can transmit the actionitem manifest to the consumer user via an electronic mail (e-mail). Inanother example, the action item identification system can notify theconsumer user regarding the action items by generating an alert on adisplay of a user device associated with the consumer user. In yetanother example, the action item identification system can provide agraphical user interface (GUI) for the consumer user to view informationregarding the action items on demand. The action item identificationsystem can analyze the conversations to identify the action itemsautomatically, e.g., based on a predefined schedule or after completionof a conversation, or on demand, e.g., based on a request from theconsumer user.

The action item identification system analyzes the features of aconversation to determine if any of the features are indicative of anaction item. The features to be analyzed could be any of the variousfeatures described above. In some embodiments, the action itemidentification system can extract features based on the usage of wordsor phrases in the conversation. For example, the action itemidentification system can be configured to identify words or phrases inthe conversation such as “I'll send you an email after the call”, “letme shoot you a proposal”, “I can run that by manager and get back toyou”, as features that are indicative of an action item. In someembodiments, the features may not be indicative of an action itemexplicitly, but implicitly, i.e., where the action item is not stated byany of the speakers explicitly. For example, the action itemidentification system can be configured to identify words or phrasessuch as “so you'll run this by your manager?” by one speaker and “Yes”by another speaker as features that are indicative of an action item(e.g. “discuss with manager”). In another example, a phrase such as “Ican do that” (where “that” refers to something previously stated) can beidentified as features that are indicative of an action item. That is,the action item identification system derives an action item from whatis expressed in previous or subsequent statements of the conversation.

The action item manifest can include the action items as one or more ofa verbatim version from the corresponding conversation or a summary ofthe action items. For example, an action item extracted verbatim fromthe conversation can be “I'll send you an email after we finish thecall,” which is generated based on the feature such as “I'll send you anemail after we finish the call.” In another example, a summarized formof the action item can be “I'll send you an email.” In some embodiments,the summarized action item can be a paraphrasing of an action item orinclude paraphrasing of an action item. For example, a paraphrasedaction item of the above same action item can be “Send email toprospect,” which is generated based on the feature such as “I'll sendyou an email after we finish the call.” The summary can be generatedusing a number of techniques, e.g., rule-based technique, semanticanalysis (e.g., parsing, noun chunking, part of speech tagging),artificial intelligence (AI), machine learning (ML) or NLP. Thesummarization or paraphrase can also include context of theconversation, e.g., speaker names, time, date, location, or a topic ofthe conversation. The context can be obtained using metadata associatedwith the conversation. An example of a summary of the action item thatincludes speaker identification information is “send quote to VP sales.”The action item manifest can also include information such as one ormore of a recording ID of one or more recordings corresponding to theconversation for which the action item is identified, a representativeID of the representative, or a customer ID of the customer involved inthe conversation or any other user identified by one of the speakersduring the first conversation, all or some of which can be obtained fromthe metadata. In some embodiments, the metadata is associated with arecording when the recording is stored at the storage system. Themetadata may be added to the recording automatically by the systemand/or can be added or modified by the consumer user.

Note that while the following paragraphs describe the action itemidentification system analyzing the features of the conversations toidentify the action items, the action item identification system is notrestricted to identifying the action items; the action itemidentification system can also identify the next steps in addition to orinstead of the action items. Further, an action item and/or a next stepcan be associated with a representative or a customer. Further, for aconversation, either or both the representative and the customer canhave an action item and/or a next step.

While the above paragraphs describe analyzing a conversation withrespect to language based features, the action item identificationsystem is not restricted to analyzing such language based features toidentify or determine an action item; the action item identificationsystem can use any feature that can be indicative of an action item. Insome embodiments, the action item identification system can determineaction items based on video features, such as facial expression or bodylanguage of the customer during the discussion of the deal. For example,the customer may ask “will you send me an email?” to which therepresentative may respond with a nod of his head. The action itemidentification system can analyze both the phrase uttered by thecustomer and the facial expression of the representative in determiningthe action item. The action item identification system can be trainedusing AI or ML techniques to extract the features from the conversationsand/or analyze the features to determine a set of features that areindicative of an action item.

Further, in some embodiments, as the action item identification systemis trained to process more conversations, e.g., using AI and MLtechniques, to extract and analyze the features, the number of featuresextracted for the same conversation can vary, e.g., more or less thanthe features that were extracted in prior analyses. This can be becausethe accuracy of the action item identification system improves withtraining, e.g., using AI and ML techniques. For example, if the actionitem identification system extracts “10” features for a conversation ata given time, it can extract more or less than “10” features for thesame conversation after it is trained to analyze “50” additionalconversations. That is, the action item identification system learns ofnew features and/or forgets old features that are not relevant toidentify action items anymore as the action item identification systemis trained to process more conversations. In some embodiments, theconsumer user can also define whether a particular feature is indicativeof an action item, and the action item identification system can furtherlearn based on the user-defined criterion, e.g., the action itemidentification system can determine, using AI and ML techniques, whatother similar or additional features are indicative of an action item.

Turning now to FIG. 1, FIG. 1 is a block diagram of a call-modelingsystem 100 in which the disclosed embodiments can be implemented. Thecall-modeling system 100 includes an offline analysis component 110 anda real-time analysis component 130. The offline analysis component 110can take as input historical call data 105, which includes conversationsbetween participants, e.g., audio recordings of calls betweenrepresentatives and customers, and process the call data 105 usingmultiple components to generate features 115 of the conversations, andclassifiers 120.

The offline analysis component 110 includes a feature generationcomponent 111 that generates features 115 by analyzing the call data 105using various techniques, e.g., ASR, NLP, AI, ML. The features 115 caninclude transcripts of audio recordings, vocabulary, semanticinformation of conversations, summarization of a call, summarizations ofutterances and various natural language entailments, voice signalassociated features (e.g., speech rate, speech volume, tone, andtimber), emotions (e.g., fear, anger, happiness, timidity, fatigue),personality traits (e.g., trustworthiness, engagement, likeability,dominance, charisma, confidence, etc.), personal attributes (e.g., age,accent, and gender), and inter-speaker attributes that indicate acomparison between both the speakers (e.g., similarity of speech ratebetween the representative and the customer, extrovert/introvertmatching, or gender or age agreement). Further, the features can includenot only aural features, but also non-aural features, e.g., visualfeatures such as body language of a participant, and facial expressionsof the participant, or any combination of aural and non-aural features.

The classifier component 112 analyzes the features 115 using varioustechniques, e.g., machine learning algorithms such as SVM, DNN, togenerate the classifiers 120. The classifiers 120 indicate conversationoutcomes, e.g., “sales closed”, “sales failed,” “probability ofrecommending to a friend,” a measure of “customer satisfaction,” and NetPromoter Score (NPS). An outcome can have binary values, e.g., “yes/no”,“high/low”, or non-binary values, e.g., a probability score, enumeratedvalues like “low, average, medium, high, very high,” values on a scaleof 0-10, etc. For example, an outcome such as customer satisfaction canbe measured using binary values such as “low/high”, or using non-binaryvalues, such as a scale of 0-10, enumerated values. Each of theclassifiers indicates a specific outcome, a probability of the specifiedoutcome and can include a set of the features that contributed to thespecific outcome. For example, in a sales call for renewing a magazinesubscription, a classifier “C1” can indicate that when laughter by acustomer and two open-ended questions from the representative areregistered, there is a high chance, e.g., 83%, of renewal.

In some embodiments, the classifier component 112 generates differentclassifiers for different time windows of the conversations. Forexample, the classifier component 112 generates a classifier “C1” forthe first two minutes of the conversations and a classifier “C2” for athird minute of the conversations. The classifier “C1” based on thefirst two minutes of the conversation can indicate that when laughter bya customer and two open-ended questions from the representative isregistered, there is a high chance, e.g., 83%, of renewal. Theclassifier “C2” based on the third minute of the conversation canindicate that when a competitor magazine or the key-phrase “read online”is used, the renewal chances drop to 10%, all of which can occur ifcustomer's speech rate drops below three words per second. Some of theclassifiers include features for inter-speaker attributes that indicatea comparison between the speakers that contribute to a specified outcome(e.g., similarity of speech rate between the representative and thecustomer, extrovert/introvert matching, or gender or age agreement).

The features, when extracted from the conversations, can includeattributes and values. The classifier determines what values of thefeatures influence a particular outcome of the call. The classifiers 120can be generated in various formats and is not limited to the aboveillustrated example format. The classifier component 112 can generatemultiple classifiers for the same outcome; however, the multipleclassifiers can have distinct sets of features. Further, as describedabove, the classifier component 112 can generate different classifiersfor different time windows of the conversation. The offline analysiscomponent 110 can store the features 115 and the classifiers 120 in astorage system 125.

The call-modeling system 100 includes a real-time analysis component 130that uses the classifiers 120 to generate on-call guidance for bothinbound and outbound calls that will help the representative optimizethe call for a desired outcome, or optimize the call duration if thedesired outcome is not predicted (i.e., very low chances of the desiredoutcome are predicted). The real-time analysis component 130 receivesreal-time call data 150 of an ongoing conversation between a customerand a representative and analyzes the real-time call data 150 togenerate a set of features, e.g., call features 135, for the ongoingconversation using a feature generation component 113. In someembodiments, the feature generation component 113 is similar to or thesame as the feature generation component 111. The feature generationcomponent 113 generates the call features 135 based on the real-timecall data 150, e.g., as described above with respect to the featuregeneration component 111. The real-time call data 150 can be anearly-stage or initial conversation between the customer and therepresentative.

After the call features 135 are generated, a classifier component 114,which, in some embodiments, is the same as, or similar to the classifiercomponent 112, inputs the call features 135 to the classifiers 120 todetermine a set of classifiers 140 that predict possible outcomes of thecall based on the call features 135. Each of the set of classifiers 140indicates a specified outcome of the call and an associated probabilityof the corresponding outcome. In some embodiments, the classifiercomponent 114 chooses classifiers that have the highest predictionpower, which can be measured using an F-score, as the set of classifiers140. After the set of classifiers 140 are determined, a call-modelingcomponent 116 generates an on-call guidance 145 that includes real-timeprobabilities of possible outcomes of the call as indicated by the setof classifiers 140. The call-modeling component 116 can further analyzethe set of classifiers 140 to determine features that have highprediction power, e.g., prediction power exceeding a specifiedthreshold, for predicting a desired outcome, and include those featuresand values associated with those features in the on-call guidance 145.The on-call guidance 145 notifies the representative to adopt, desist orpersist with an on-call behavior consistent with those features toachieve the desired outcome, or to increase the probability of achievingthe desired outcome. If the set of classifiers 140 predict that thedesired outcome may not be achieved, the call-modeling component 116 maysuggest, in the on-call guidance 145, that the representative wrap upthe call.

The call data 105 can be in various formats, e.g., audio recordings,transcripts of audio recordings, online chat conversations. Similarly,the real-time call data 150 can be in various formats, e.g., real-timeaudio stream of the call, a chat transcript of an ongoing conversationin an online chat application. Further, the real-time call data 150,which can include an initial or early stage conversation, can be aconversation between the customer and an automated machine, e.g., aninteractive voice response (IVR) system, or a representative forgathering preliminary information from the customer that can be usefulfor generating the on-call guidance.

In some embodiments, the call-modeling system 100 includes a search toolthat empowers a consumer user to explore various aspects of aconversation. For example, the search tool allows the consumer user tosearch for anything that came up on the call, e.g., both linguistic andmeta-linguistic. The search tool can be used to further analyze theconversation, extract appropriate features and use them to improve theclassifiers in predicting the outcome of the calls. For example, thesearch tool can be used to find calls that registered a laughter fromthe customer, calls in which the customer spoke for the first time aftera specified number of minutes, calls in which the customer soundedangry, calls in which customer mentioned competitors, calls in which therepresentatives engaged in rapport building, calls in which therepresentative modulated speech rates at various instances of the call,calls in which short or open-ended questions were asked at a highfrequency, or any combination of the above.

FIG. 2 is a block diagram of a feature generation component of FIG. 1for extracting features from call data, consistent with variousembodiments. In some embodiments, the feature generation component 111includes an ASR component 210, an NLP component 225, an affect component215 and a metadata component 220. The ASR component 210 can analyze calldata 205, e.g., a voice recording, and produce a transcription,vocabulary, and a language model of the conversation. The NLP component225 can extract semantic information, such as key objection handlingresponses, from the output of the ASR component 210. The affectcomponent 215 can analyze the call data 205 for emotional signals andpersonality traits (e.g., likeability, extroversion/introversion,charisma, confidence, and trustworthiness) as well as general personalattributes such as gender, age, and accent of the participants. Themetadata component 220 can extract data regarding conversation flow(e.g., who spoke when, and how much silence and overlap occurred). Insome embodiments, the above components can process the call data 105 inparallel. The output of the components can be generated as features 115of the conversations, which can be analyzed to determine outcomes of theconversations.

The ASR component 210 may be tuned for specific applications, e.g., forsales calls. The features produced by the ASR component 210 may includefull transcripts, vocabularies, statistical language models (e.g.,transition probabilities), histograms of word occurrences (“bag ofwords”), weighted histograms (where words are weighted according totheir contextual salience, using e.g., a Term Frequency—Inverse DocumentFrequency (TF-IDF) scheme), n-best results, or any other data availablefrom the component's lattice, such as phoneme time-stamps, etc. The ASRcomponent 210 may also be used to extract meta-linguistic features suchas laughter, hesitation, gasping, background noise, etc. The ASRfeatures can be extracted separately for the representative and thecustomer, and may be recorded separately for multiple speakers on eachside of the conversation.

The NLP component 225 processes the text to produce various semanticfeatures, e.g., identification of topics, identification of open-endedquestions, identification of objections and their correlation withspecific questions, named entity recognition (NER), identification ofrelations between entities, identification of competitors and/orproducts, identification of key phrases and keywords (eitherpredetermined, or identified using salience heuristics such as TF-IDF),etc. Additional features that may be extracted by the NLP component 225can be summarizations of utterances and various natural languageentailments. The NLP features can be extracted separately for therepresentative and the customer, and may be recorded separately formultiple speakers on each side of the conversation.

The affect component 215 can extract low-level features and high-levelfeatures. The low-level features can refer to the voice signal itselfand can include features such as speech rate, speech volume, tone,timber, range of pitch, as well as any statistical data over suchfeatures (e.g., maximal speech rate, mean volume, duration of speechover given pitch, standard deviation of pitch range, etc.). Thehigh-level features can refer to learned abstractions and can includeidentified emotions (e.g., fear, anger, happiness, timidity, fatigue,etc.) as well as perceived personality traits (e.g., trustworthiness,engagement, likeability, dominance, charisma, confidence, etc.) andperceived or absolute personal attributes such as age, accent, andgender. Emotion identification, personality trait identification, andpersonal attributes, may be trained independently to produce modelsincorporated by the affect component, or trained using the humanjudgment tags optionally provided to the offline analysis component. Insome embodiments, the affect component 215 can also extract features,such as a speaker engagement metric (“wow” metric), which measures howengaged a participant was in the conversation, e.g., based on the usageof vocabulary, rate of speech, pitch change. For example, the usage ofphrase “Oh! cool” can indicate a higher degree of engagement than thephrase “cool!”. In another example, the same phrase but said indifferent pitches or pitch ranges can indicate different degrees ofengagement. All features extracted by the affect component 215 may ormay not include a corresponding confidence level, which can be used inmodeling outcomes. The affect features can be extracted separately forthe representative and the customer, and may be recorded separately formultiple speakers on each side of the conversation.

The metadata component 220 can measure conversation flow, includingspeaker diarisation (e.g., which speaker spoke when and for how long),silence times and duration, as well as overlap of two or more speakersin addition to other metadata such as time of day call was placed,geographical destination of call and known gender and age ofparticipants. The data extracted with the metadata component 220 may becollected separately for multiple speakers on each side of theconversation, or pooled together for representative and customer sides,respectively.

All components may extract features for a group of representatives, asingle representative and/or a customer, including multiple parties oneither side, and may be customized to optimize feature extractionaccordingly. In addition, the features 115 may be extracted on therepresentative's recording alone, on the customer's recording alone, oron both. The features 115 may also include comparisons between extractedattributes. For example, the affect component 215 may extract as afeature a mean difference between representative and customer's speechrates, or a maximum difference between representative and customer'sspeech pitches. Likewise, the ASR component 210 may extracttranscriptions and keywords both as a combined transcript and as twoseparate transcripts, and may be tuned with an acoustic or languagemodel specific to a group of representatives or an individualrepresentative. Similarly, the NLP component 225 may extract featuressuch as open-ended questions with or without the corresponding response.

In some embodiments, the feature generation component 111 can alsogenerate a set of features that indicate a blueprint of a conversation.The blueprint can represent a skeleton of the conversation and indicatea presence or absence of a particular aspect in the conversation. Forexample, the blueprint can include various features that indicatewhether the conversation included any agenda setting, rapport building,clarification questions, defining goals, setting expectations,mentioning of examples. The blueprint can also help in predictiveanalysis of the outcome of the calls, e.g., by the classifier component112. One or more components of the feature generation component 111 canuse AL and/or ML techniques to extract one or more of the features 115.

FIG. 3 is a block diagram of the classifier component for generatingclassifiers, consistent with various embodiments. The example 300illustrates the classifier component 112 using the features 115extracted from the feature generation component 111 to generate a numberof classifiers, “C1”-“CN”. In some embodiments, the classifier component112 analyzes the features of a dedicated portion of the collectedrecordings, e.g., a training set, which is a subset of the entirerecordings available for analysis, to generate the classifiers 120. Eachof the classifiers 120 can have a value, e.g., an F-score, thatindicates a prediction power of the classifier for the specifiedoutcome. The higher the prediction power, the higher the probability ofachieving the specified outcome of the classifier based on the includedfeatures. In some embodiments, the prediction power is determined byrunning the classifiers 120 on, e.g., a portion of call recordings thatis not yet analyzed, e.g., a test set, and computing the respectiveF-score.

The classifiers 120 may be further analyzed to determine what featurescarry the largest prediction power, e.g., speech rate early in theconversation, occurrence of first interrupt by customer, names ofcompetitors mentioned, or number of open questions thoughtfullyanswered, and a subset of these classifiers that have features with thelargest prediction power can be used to generate the on-call guidance.

The conversation outcome depicted by the classifiers 120 can be anyconfigurable outcome, e.g., “sales closed”, “sales failed”, “demoscheduled”, “follow up requested,” NPS-like probability of recommendingto a friend, etc. In some embodiments, the features 115 extracted fromthe feature generation component 111 can be fed into a machine learningalgorithm (e.g., a linear classifier, such as a SVM, or a non-linearalgorithm, such as a DNN or one of its variants) to produce theclassifiers 120. The classifiers may be further analyzed to determinewhat features carry the largest prediction powers (e.g., similarity ofspeech rate, occurrence of first interrupt by customer,extrovert/introvert matching, or gender or age agreement.)

The classifier component 112 can generate multiple classifiers for thesame outcome. However, for a given outcome, different classifiers havedifferent features. For example, the classifier component 112 cangenerate a first classifier 305, “C1,” and a second classifier 310,“C2,” for a specified outcome, “O1.” However, the first classifier “C1”has a first set of features, e.g., features “f1”-“f3,” and the secondclassifier “C2” has a second set of features, e.g., features “f5”-“f8.”The features in different classifiers can have different predictionpowers and contribute to the specified outcome in different degrees.

Different classifiers may be built for a different number ofparticipants, and may consider multiple participants as a singleinterlocutor, or as distinct entities. Further, as described above, theclassifier component 112 can generate different classifiers fordifferent time intervals of a conversation. The classifier component 112analyzes the features 115 extracted from the feature generationcomponent 111 at various time intervals, e.g., seconds 00:05-00:10,seconds 00:20-00:30, minutes 1:00-2:00, covering the entire conversationduration, and generates one or more classifiers for each of those timeintervals. Each classifier can correspond to a specified time intervalof the conversation. For example, if “100” conversations are beinganalyzed, then the classifier component 112 can analyze first 5-20seconds each of the “100” conversations and generate one or moreclassifiers for all the conversations corresponding to the interval of5-20 seconds. Similarly, it can generate one or more classifierscorresponding to the 10-25 seconds interval. If more than one classifieris generated for a specified time interval, in some embodiments,different classifiers can have different outcomes, and in someembodiments, can have the same outcome; however, different classifierswill have different sets of features that contribute to thecorresponding outcome. In the example 300, classifiers C1 and C5correspond to a time window of seconds 00:05-00:20 of the conversationsanalyzed, and classifier C10 corresponds to minute 1:00-2:00 of theconversations.

The classifier 315, “C3,” includes an example set of features extractedfrom analyzing various sales calls. The classifier 315 corresponds tothe first two minutes of the conversations, and indicates that whenlaughter from the customer is registered and the representative greetsthe customer, indulges in rapport building and poses at least twoopen-ended questions, then there is a high chance, e.g., 83%, of renewalof a magazine subscription. The features and outcome of the classifier315 “C3” can be “f1→customer laughter=yes” “f2→greeting customer=yes,”“f3→rapport building=yes,” (“f4→open ended questions asked=yes,” and“f5→number of open ended questions asked=2”), “outcome=renewsubscription” “probability of outcome=83%.”

The classifiers 120 can be used by the real-time analysis component 130,e.g., as described at least with reference to FIG. 1 above and FIG. 4below, to generate an on-call guidance for representatives or bothinbound and outbound calls. FIG. 4 is a block diagram of the real-timeanalysis component of FIG. 1 for generating on-call guidance for arepresentative during a call between the representative and a customer,consistent with various embodiments. In some embodiments, the real-timeanalysis component 130 takes as input a live conversation stream, e.g.,real-time call data 150, between a representative 410 and a customer405, uses the feature generation component 113 to extract call features135, e.g., as described above at least with reference to FIGS. 1 and 3.

The classifier component 114 feeds the call features 135 into theclassifiers 120 generated by the offline analysis component 110 andselects a subset of the classifiers 120, e.g., a set of classifiers 140,that includes features that match with the call features 135 extractedfrom the live conversation stream. In some embodiments, the set ofclassifiers 140 chosen by the call-modeling component 116 are also theclassifiers that have high predictability power, e.g., as measured usingan F-score and that have an F-score exceeding a specified threshold.

The call-modeling component 116 then generates an on-call guidance 145,which includes information regarding real-time probabilities forspecific outcomes to which the set of classifiers 140 correspond. Theon-call guidance 145 may be used to notify the representative and/ortheir managers of the predicted outcome of the call. Additionally, thecall-modeling component 116 can further analyze the set of classifiers140 to determine classifiers that include features with the largestprediction powers, and present the values of those features in theon-call guidance 145 for suggesting the representative and/or theirmanagers to modify or persist with an on-call behavior consistent withthose features. For example, if one of the set of classifiers 140predicts that conversations with rapport building and several open-endedquestions being posed at the first few minutes of the conversation leadto favorable outcomes, the call-modeling component 116 may notify therepresentative and/or their managers as part of the on-call guidance 145to engage in rapport building and pose questions at early stages of theconversation. Similarly, if one of the classifiers from the set ofclassifiers 140 indicates that matching speech rate to within 10% ofcustomer's rate at a specified relative position of the call, e.g.,during third minute of the call, leads to improved closing results, thecall-modeling component 116 may notify the representative and/or theirmanagers as part of the on-call guidance 145 to adjust their speech rateaccordingly. On the other hand, if one of the classifiers from the setof classifiers 140 indicates that conversations beginning with over aspecified number of objections, naming a specific competitor and mentionof the phrase “read online” do not lead to good results, thecall-modeling component 116 may notify the representative and/or theirmanagers as part of the on-call guidance 145 to expedite wrap-up ofconversations to avoid losing time on a call that is not likely to yielddesired results.

In addition to live on-call guidance, the real-time analysis component130 may be used to provide the representative and/or their managers withnon-real time analysis as well, which provides insight into details ofthe conversations, e.g., what occurred in the conversations, when eventsoccurred, and various such quantifiable analytics of the calls. Forexample, the classifiers can be used to find interesting calls thatwould interest the representatives to listen and learn from. Thedisclosed embodiments can be used to improve outcomes of the call notonly during a real-time or a live call, but could also be used to informrepresentatives and/or managers for better training and coaching inretrospect.

The real-time analysis component 130 may also be used to auto-populateinformation fields in a customer relationship management (CRM) system ora similar system.

FIG. 5 is a flow diagram of a process 500 for performing offlineanalysis of conversations between participants, consistent with variousembodiments. In some embodiments, the process 500 can be implemented inthe call-modeling system 100 of FIG. 1. At block 505, the offlineanalysis component 110 retrieves historical call data, e.g., call data105, regarding various conversations between participants, such as acustomer and a representative. In some embodiments, the call data 105can be audio recordings of calls between the participants, transcriptsof audio recordings, chat transcripts, etc. The offline analysiscomponent 110 can retrieve the call data 105 from the storage system125. Further, in some embodiments, the call data 105 can include dataregarding only a subset of the conversations stored in the storagesystem 125.

At block 510, the feature generation component 111 analyzes the calldata 105 to extract various features of the conversation, e.g., asdescribed at least with reference to FIGS. 1 and 2. Some examplefeatures include transcripts of audio recordings, vocabulary, semanticinformation of conversations, summarizations of utterances and variousnatural language entailments, voice signal associated features (e.g.,speech rate, speech volume, tone, and timber), emotions (e.g., fear,anger, happiness, timidity, fatigue), inter-speaker features (e.g.,similarity of speech rate between speakers, occurrence of firstinterrupt by customer, extrovert/introvert matching, or gender or ageagreement), personality traits (e.g., trustworthiness, engagement,likeability, dominance, charisma, confidence, etc.) and personalattributes (e.g., age, accent, and gender). The feature generationcomponent 111 can also analyze the call data 105 to generate varioustags as described above.

At block 515, the classifier component 112 analyzes the features togenerate classifiers, e.g., as described at least with reference toFIGS. 1 and 3. The classifier component 112 analyzes the features 115using various techniques, e.g., machine learning algorithms such as SVM,DNN, to generate the classifiers 120. The classifiers 120 indicateconversation outcomes, e.g., “sales closed”, “sales failed,”“probability of recommending to a friend,” NPS, or customersatisfaction. Each of the classifiers indicates a specific outcome andcan include a set of the features that contributed to the specificoutcome. For example, in a sales call for renewing a magazinesubscription, a classifier “C1” can indicate that when laughter by acustomer and two open-ended questions from the representative areregistered, there is a high chance, e.g., 83%, of renewal. Theclassifier component 112 can generate multiple classifiers for the sameoutcome; however, they have distinct sets of features. Further, theclassifier component 112 generates different classifiers for differenttime windows of the conversations. For example, the classifier component112 generates a classifier “C1” for first two minutes of theconversations and a classifier “C2” for a third minute of theconversations. The offline analysis component 110 can store the features115 and the classifiers 120 in a storage system 125.

FIG. 6 is a flow diagram of a process 600 for modeling calls betweenparticipants to generate on-call guidance, consistent with variousembodiments. In some embodiments, the process 600 can be implemented inthe call-modeling system 100 of FIG. 1. At block 605, the real-timeanalysis component 130 receives real-time call data 150 of an ongoingconversation, e.g., an audio stream of a voice call between a customerand a representative. At block 610, the feature generation component 113analyzes the real-time call data 150 to extract features, e.g., callfeatures 135, of the ongoing conversation, e.g., as described at leastwith reference to FIGS. 1 and 2. The feature generation component 113can also analyze the real-time call data 150 to generate various tags asdescribed above.

At block 615, the classifier component 114 inputs the extracted featuresto classifiers in the storage system, e.g., classifiers 120 which aregenerated as described at least with reference to process 500 of FIG. 5,to determine one or more classifiers that predict possible outcomes ofthe call based on the extracted features. For example, as described atleast with reference to FIGS. 1 and 4, the classifier component 114feeds the extracted features 135 into the classifiers 120 generated bythe offline analysis component 110, and selects a subset of theclassifiers 120, e.g., a set of classifiers 140, that includes featuresthat match with the call features 135 extracted from the liveconversation stream. In some embodiments, the set of classifiers 140include classifiers whose prediction power exceeds a specifiedthreshold. The set of classifiers 140 corresponds to specific outcomesand include real-time probabilities for the specific outcomes.

At block 620, the call-modeling component 116 generates on-callguidance, e.g., on-call guidance 145, that presents the real-timeprobabilities of possible outcomes of the call as indicated by the setof classifiers 140. The call-modeling component 116 can further analyzethe set of classifiers 140 to determine features that have highprediction power, e.g., prediction power exceeding a specifiedthreshold, for predicting a desired outcome, and then include thosefeatures and values associated with those features in the on-callguidance 145. The on-call guidance 145 notifies the representative toadopt or persist with an on-call behavior consistent with those featuresto achieve the desired outcome, or at least to increase the probabilityof achieving the desired outcome. For example, the on-call guidance 145can present instructions on a display screen of a user device associatedwith the representative recommending the representative to change therate of speech, use specific key words, or pose more open-endedquestions to the customer in order to increase the probability ofachieving the desired outcome.

Example Usage of the Embodiments

The following is an example usage of the disclosed embodiments formodeling sales calls for renewal of a subscription for a magazine. At afirst stage, e.g., before a call is received from a live customer orbefore a call is placed by a representative, a number of recordings ofprevious calls is processed by the offline analysis component 110, e.g.,using an ASR component 210 that is customized for the field of surgeryinstitutions, an NLP component 225, an affect component 215 and ametadata component 220 to generate various features. The classifiercomponent 112 generates two classifiers, based on those features, thatcan be found to be highly predictive: (a) a first classifier based onthe first two minutes of one or more of the analyzed conversations,which indicates that when a laughter by the customer is registered, therepresentative engages in rapport building, and at least two open-endedquestions are posed by the representative, then there is a high chance,e.g., 83%, of subscription renewal; (b) a second classifier based on thethird minute of one or more of the analyzed conversations, whichindicates that when a competitor magazine or the key-phrase “readonline” is used, and/or the speech rate of the customer is more thanthree words per second, the renewal chances drop to 10%.

The above two classifiers can then be used by the real-time analysiscomponent 130 in a second stage, e.g., during a live call between therepresentative and the customer, for generating an on-call guidance toguide the sales representatives as follows. Based on the firstclassifier, the real-time analysis component 130 can indicate to thesales representative to ask questions within the first 2 minutes. Basedon the second classifier, the real-time analysis component 130 can, atminute three of the conversation, urge the representative to reducespeech rate to get the customer to mirror their own speech rate if acompetitor is mentioned or otherwise the phrase “read online” is used.If the speech rate is not reduced, the real-time analysis component 130can indicate to the representative and/or their managers to wrap up thecall as soon as possible.

The embodiments disclosed above may be implemented as separate modules,e.g., as presented above, as a single module, or any combinationthereof. Implementation details may vary, including core machinelearning algorithms employed. The embodiments may be implemented usingany software development environment or computer language. Theembodiments may be provided as a packaged software product, aweb-service, an API or any other means of software service. Theembodiments may use expert taggers, crowdsourcing or a hybrid approachfor tagging.

FIG. 7 is a block diagram of an action item identification system,consistent with various embodiments. The action item identificationsystem 700 can analyze the call data 105 to determine action items andnext steps resulting from one or more of the conversations, and generateinformation regarding such action items and next steps, e.g., in anaction item manifest 725. A consumer user can use the action itemmanifest 725 for various purposes. For example, the consumer user canuse the action item manifest 725 for analysis of behaviors of therepresentative and/or the customer in the conversation. In anotherexample, the representative or the customer can review the action itemsin the action item manifest 725 and initiate an appropriate actionaccordingly. In yet another example, a consumer user such as asupervisor of the representatives can keep track of the action items ofvarious representatives and ensure that the representatives areattending to those action items.

The action item identification system 700 includes a feature generationcomponent, such as feature generation component 111 of FIG. 1, thatgenerates features 115 of the conversations by analyzing the call data105 stored in the storage system 125. The call data 105 can include anumber of recordings, such as a first recording 730 of a firstconversation between a first representative and a first customer of theorganization (e.g., for discussing a first deal the organization isoffering to the first customer), and a second recording 731 of a secondconversation between a second representative and a second customer(e.g., for discussing a second deal the organization is offering to thesecond customer). The feature generation component 111 analyzes thefirst recording 730 to extract the first set of features 735 of thefirst conversation and the second recording 731 to extract the secondset of features 736 of the second conversation. In some embodiments, thefirst set of features 735 and the second set of features 736 are asubset of the features 115. Note that the first representative can havemultiple conversations with the first customer, e.g., to discuss thefirst deal, and can have conversations with different customers, e.g.,to discuss different deals, and therefore, different conversations ofthe first representative can have different action items. In someembodiments, a recording can be tagged with metadata, such as therepresentative ID, the customer ID and the deal ID of a deal beingdiscussed in the conversation, all or some of which can identify thecontext of the conversation. Note that the recordings can be of aconversation that is any of telephone based, VoIP based, videoconference based, VR based, AR based, e-mail based, or in-personinteraction based.

The first set of features 735 can include transcripts of theconversations, vocabulary, semantic information of conversations,summarization of a call, summarizations of utterances and variousnatural language entailments, voice signal associated features (e.g.,speech rate, speech volume, tone, and timber), detected emotions (e.g.,fear, anger, happiness, timidity, fatigue, laughter), detectedpersonality traits (e.g., trustworthiness, engagement, likeability,dominance, charisma, confidence, etc.), personal attributes (e.g., age,accent, and gender), and inter-speaker attributes that indicate acomparison between both the participants (e.g., similarity of speechrate between the representative and the customer, extrovert/introvertmatching, or gender or age agreement). The first set of features 735 caninclude usage of words or phrases features such as a specific word,phrase, and pronouns. The first set of features 735 can also include anyof length of utterances and/or turns taken by a participant in talkingduring the conversation, talk-to-listen ratio of a representative or acustomer, or any other behavioral feature of the customer. The first setof features 735 can be features associated with the first customer, thefirst representative, the conversation, or a combination. The first setof features 735 can also include information that indicates with whichparticipant a particular feature is associated.

Further, the first set of features 735 can include not only auralfeatures, but also non-aural features, e.g., visual features such asbody language of a participant, and facial expressions of theparticipant, or any combination of aural and non-aural features. One ormore features from the first set of features 735 could also be generatedfrom the transcripts of any of emails, online messages, and onlinemeetings. In some embodiments, the feature generation component 111 candetermine that any of a word, a phrase, a text, emoji, symbols, or acombination thereof can convey a particular feature. For example, thefeature generation component 111 can determine that a text such as “HaHa” or “rofl” in the transcript can indicate laughter. In someembodiments, the second set of features 736 includes features similar tothe first set of features 735.

In analyzing the first set of features 735 to determine if the firstconversation includes any action items, the action item recognitioncomponent 705 determines if any of the first set of features 735satisfies the criterion for being indicative of an action item. In someembodiments, the criterion can be specified by the consumer user, and/orlearnt by the action item recognition component 705, e.g., using AI andML techniques, based on the criteria defined by the consumer user. Forexample, if the features based on which the action items are to beidentified are usage of words or phrases in the first conversation, thenthe criteria for the words or phrases to be indicative of the actionitem can be defined by the consumer user, and/or the action itemrecognition component 705 can be trained using AI and ML techniques torecognize the words or phrases that are indicative of the action item.The action item recognition component 705 can analyze those of the firstset of features 735 that correspond to the usage of words or phrases inthe first conversation and identify words or phrases such as “I'll sendyou an email after the call”, “let me shoot you a proposal”, or “I canrun that by manager and get back to you” to be indicative of an actionitem. In some embodiments, the action item recognition component 705 canalso identify words or phrases that are implicitly indicative, i.e.where the action item is not explicitly stated by the speaker, of anaction item. For example, phrases such as:

Speaker 1: “so you'll run this by your manager?”

Speaker 2: “ . . . Yes”,

can be determined to be indicative of an action item. In anotherexample, a phrase such as “I can do that” (where “that” refers tosomething previously stated) can also be determined to be indicative ofan action item.

The action item recognition component 705 can use AI, ML, process-driventechniques (e.g., programmed by the consumer user) or a combination toidentify the features (e.g., the above features, other such similarfeatures or additional features) that are indicative of the action item.

Similar to the action items, the action item recognition component 705can also identify words or phrases in the first conversation that areindicative of next steps. For example, the action item recognitioncomponent 705 can identify a phrase such as “I will wait for your call,”as indicative of the next step—“wait for call.” In another example, theaction item recognition component 705 can identify phrases that areimplicitly indicative of the next step—“Speaker 1: ‘I will send you anemail,’ and Speaker 2: ‘ok’” as features that are indicative of the nextstep—“wait for email” for “Speaker 2.”

The action item recognition component 705 is not restricted to the aboveconversational language-based features for determining features that areindicative of an action item. The action item recognition component 705can use any feature associated with the first conversation that can beindicative of the action item. In some embodiments, the action itemrecognition component 705 determines the action items based on videofeatures, such as facial expression or body language of the customerduring the discussion of the deal; based on voice signal associatedfeatures (e.g., a speech rate, a speech volume, a tone, and a timber) ofthe customer, emotions of the customer (e.g., fear, anger, happiness,timidity, fatigue), personality traits of the customer (e.g.,trustworthiness, engagement, likeability, dominance, etc.), ortalk-listen ratio of the customer. For example, the customer may ask“will you send me an email?” to which the representative may respondwith a nod of his head. The action item recognition component 705 cananalyze both the phrase uttered by the customer—“will you send me anemail?”—and the facial expression of the representative—a nod of hishead—in determining the features that are indicative of the action item.

Upon determining the features that are indicative of an action itemassociated with the first conversation, the action item recognitioncomponent 705 can generate the action item data 710, which includes oneor more features that are indicative of the action item, e.g., one ormore words or phrases in the first conversation that are determined tobe indicative of an action item. Similarly, the action item recognitioncomponent 705 analyzes the first set of features 735 to determine if anyfeature from the first set of features 735 is indicative of a next step,and if so, generates next step data 715 that includes one or morefeatures that are indicative of the next step.

The action item recognition component 705 can similarly analyze variousother recordings in the storage system 125, and generate action itemdata 710 for one or more conversations. The action item recognitioncomponent 705 can generate separate action item data objects for each ofthe conversations, or generate a single action item data object thatincludes features indicative of action items for a number ofconversations, e.g., multiple conversations of a representative ormultiple conversations of a representative-customer pair.

The action item generation component 720 analyzes the features in theaction item data 710 and generates the action items for the firstconversation in the action item manifest 725. The action item generationcomponent 720 can generate the action items verbatim from what isdiscussed in the conversations, or generate a summary of the identifiedaction items. Examples of various forms of action items generated in theaction item manifest 725 are described at least in association with FIG.8.

FIG. 8 illustrates various examples 800 of action items identified bythe action item identification system, consistent with variousembodiments. The action item generation component 720 can generate anaction item verbatim, e.g., using the same words or phrases from thefirst conversation that were determined to be indicative of the actionitem. For example, if the action item data 710 has a feature—“I'll sendyou an email after we finish the call” as indicative of the action item,the action item generation component 720 can generate a first actionitem 805 as “I'll send you an email after we finish the call,” which isverbatim from the first conversation.

In some embodiments, the action item generation component 720 summarizesthe action item in the action item manifest 725 based on the features inthe action item data 710. For example, if the action item data 710 has afeature—“I'll send you an email after we finish the call” as indicativeof the action item, the action item generation component 720 cangenerate a summarized action item 810 as “Send email to prospect.” Thesummary can be generated using a number of techniques, e.g., rule-basedtechnique, semantic analysis (e.g., parsing, noun chunking, part ofspeech tagging), AI, ML or NLP. In some embodiments, the summarizationcan also include context of the first conversation, e.g., speaker names,time, date, location, or a topic of the first conversation. The contextcan be obtained using metadata associated with the first conversation.For example, the action item generation component 720 can generate thesummarized action item 810 that includes speaker identificationinformation as “Send quote to VP of Sales,” where “VP of sales” is thefirst customer associated with the first conversation or any other useridentified by one of the speakers during the first conversation.

Similar to the action items, the action item generation component 720can also generate the next steps in the action item manifest 725 basedon the next step data 715. For example, if the next step data 715 has afeature—“I will wait for your email” as indicative of the next step, theaction item generation component 720 can generate a first next step 815as “I will wait for your email,” which is verbatim from the firstconversation. In some embodiments, the action item generation component720 summarizes the next step in the action item manifest 725 based onthe features in the next step data 715. For example, if the next stepdata 715 has a feature—“I will wait for your email” as indicative of theaction item, the action item generation component 720 can generate asummarized next step 820 as “Wait for email from vendor,” in which thesummarization is performed based on a context of the first conversation.

FIG. 9 is an example of action item data having features that areindicative of an action item, consistent with various embodiments. Asdescribed above at least with reference to FIG. 7, the action itemgeneration component identifies the features of the first conversationthat are indicative of an action item and includes them in the actionitem data 710. The features included in the action item data 710 can beindicative of the action item explicitly or implicitly. For example, theaction item generation component 720 identifies a first feature 905—“Ican run that by manager and get back to you” as feature that isindicative of an action item explicitly. The action item generationcomponent 720 can generate the action item corresponding to the firstfeature 905 verbatim or summarize the first feature 905, e.g., asdescribed at least with reference to FIG. 8.

In another example, the action item generation component 720 identifiesa second feature 910—“Speaker 1: ‘so you'll run this by your manager?’and Speaker 2: ‘Yes’” as a feature that is indicative of an action itemimplicitly. In some embodiments, the features that are indicative of theaction item implicitly may not provide context associated with theaction item if included in the action item manifest 725 verbatim.Accordingly, the action item generation component 720 generates theaction item by summarizing the second feature 910, e.g., to “Verifyquote with manager and email the quote to prospect.” For summarizing afeature, the action item recognition component 705 may have to analyzethe first conversation prior to and/or after the feature occurred in thefirst conversation to obtain the context. For example, to summarize“this” of “you'll run this by . . . ” to “quote,” the action itemrecognition component 705 may have to analyze the first conversationprior to the feature “you'll run this by . . . ” and include the portionof the conversation having the feature “Give me your best quote” in theaction item data 710 for the action item generation component 720 toobtain the context “quote.” The action item generation component 720 cangenerate the summary using techniques described at least with referenceto FIG. 8.

In another example, the action item generation component 720 identifiesa third feature 915—“Speaker 1: ‘If you email me the quote, I'll checkwith my manager and get back to you?’ and Speaker 2: ‘I can do that” asa feature that is indicative of an action item implicitly. The actionitem generation component 720 generates the action item in the actionitem manifest 725 by summarizing the third feature 915, e.g., to “Emailquote to the prospect.” As describe above, for summarizing a feature,the action item recognition component 705 may have to analyze the firstconversation prior to and/or after the feature “I can do that” occurredin the first conversation to obtain the context of “that” as “quote”(not illustrated in figure).

Referring back to action item generation component 720 of FIG. 7, theaction item generation component 720 analyzes the action item data 710and the next step data 715, and generates the action item manifest 725having the action items and the next steps, as described above at leastwith reference to FIGS. 8 and 9. An action item and a next step in theaction item manifest 725 can be associated with any entity, e.g., therepresentative or the customer. The action item manifest 725 canidentify which action item is associated with which entity. In someembodiments, for a given conversation, the action item generationcomponent 720 can generate one action item manifest 725, which includesaction items associated with both the representative and the customer.In some embodiments, for a given conversation, the action itemgeneration component 720 can generate two separate action itemmanifests, a first one including action items associated with therepresentative and a second one including action items associated withthe customer.

Further, the action item generation component 720 can generate an actionitem manifest 725 such that the action items and next steps in theaction item manifest 725 are for a single conversation, multipleconversations of a representative, multiple conversations of arepresentative-customer pair, multiple conversations of arepresentative-customer-deal triplet, or multiple conversationsassociated with representatives of a particular division or role in theorganization. The action item manifest 725 can be shared with one ormore consumer users, e.g., via email.

FIG. 10 is a flow diagram of a process 1000 for determining action itemsby analyzing conversations of representatives, consistent with variousembodiments. In some embodiments, the process 1000 can be implementedusing the action item identification system 700 of FIG. 7. At block1005, the feature generation component 110 retrieves recordings ofconversations of representatives, e.g., a first recording 730 and asecond recording 731. Each of the recordings is of a conversationbetween at least one of the representatives and one of the customers.

At block 1010, the feature generation component 111 extracts featuresfrom the recordings, e.g., the first set of features 735 correspondingto the first recording 730 and second set of features 736 correspondingto the second recording. The first set of features 735 can indicatecharacteristics of any of (a) a first customer in a correspondingconversation, (b) a first representative in the correspondingconversation, and/or (c) the corresponding conversation. Similarly, thesecond set of features 736 can indicate characteristics of any of (a) asecond customer in a corresponding conversation, (b) a secondrepresentative in the corresponding conversation, or (c) thecorresponding conversation.

At block 1015, the action item recognition component 705 analyzes thefeatures of each of the conversations to determine a set of featuresthat is indicative of an action item associated with the correspondingconversation, e.g., as described at least with reference to FIG. 7. Forexample, the action item identification system can be configured toidentify words or phrases in the conversation such as “I'll send you anemail after the call”, “let me shoot you a proposal”, “I can run that bymanager and get back to you”, as features that are indicative of anaction item. The action item recognition component 705 can determinefeatures that are indicative of an action item explicitly or implicitly.The action item recognition component 705 stores the set of featuresthat is indicative of the action item in an action item data object 710.Similarly, the action item recognition component 705 determines featuresthat are indicative of next steps, and stores them in a next step dataobject 715. Additional details with respect to generating the actionitem data and the next step data are described at least with referenceto FIGS. 7-9.

At block 1020, the action item generation component 720 analyzes the setof features in the action item data 710 and generates the action itemsin an action item manifest 725 based on the set of features, e.g., asdescribed at least with reference to FIGS. 7-9. The action itemgeneration component 720 can generate the action items verbatim fromwhat is discussed in the conversations, or generate a summary of theidentified action items.

Further, the action item identification system 700 can be used toanalyze recordings of calls (e.g., offline mode) as described above,and/or can be used to analyze a real-time call or an ongoing call (e.g.,online mode) between a representative and the customer and notify therepresentative, e.g., on a screen of a computing device associated withthe representative or any other consumer user, an action item that isidentified by the action item identification system 700 during the call.The action item identification system 700 can perform a similar analysiswith the ongoing call as with the recordings of the call.

FIG. 11 is a block diagram of a computer system as may be used toimplement features of the disclosed embodiments. The computing system1100 may be used to implement any of the entities, components orservices depicted in the examples of the foregoing figures (and anyother components described in this specification). The computing system1100 may include one or more central processing units (“processors”)1105, memory 1110, input/output devices 1125 (e.g., keyboard andpointing devices, display devices), storage devices 1120 (e.g., diskdrives), and network adapters 1130 (e.g., network interfaces) that areconnected to an interconnect 1115. The interconnect 1115 is illustratedas an abstraction that represents any one or more separate physicalbuses, point to point connections, or both connected by appropriatebridges, adapters, or controllers. The interconnect 1115, therefore, mayinclude, for example, a system bus, a Peripheral Component Interconnect(PCI) bus or PCI-Express bus, a HyperTransport or industry standardarchitecture (ISA) bus, a small computer system interface (SCSI) bus, auniversal serial bus (USB), IIC (I2C) bus, or an Institute of Electricaland Electronics Engineers (IEEE) standard 1394 bus, also called“Firewire”.

The memory 1110 and storage devices 1120 are computer-readable storagemedia that may store instructions that implement at least portions ofthe described embodiments. In addition, the data structures and messagestructures may be stored or transmitted via a data transmission medium,such as a signal on a communications link. Various communications linksmay be used, such as the Internet, a local area network, a wide areanetwork, or a point-to-point dial-up connection. Thus, computer readablemedia can include computer-readable storage media (e.g.,“non-transitory” media) and computer-readable transmission media.

The instructions stored in memory 1110 can be implemented as softwareand/or firmware to program the processor(s) 1105 to carry out actionsdescribed above. In some embodiments, such software or firmware may beinitially provided to the processing system 1100 by downloading it froma remote system through the computing system 1100 (e.g., via networkadapter 1130).

The embodiments introduced herein can be implemented by, for example,programmable circuitry (e.g., one or more microprocessors) programmedwith software and/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more ASICs, PLDs, FPGAs, etc.

Remarks

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding of the disclosure. However, in someinstances, well-known details are not described to avoid obscuring thedescription. Further, various modifications may be made withoutdeviating from the scope of the embodiments. Accordingly, theembodiments are not limited except as by the appended claims.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described, which may be requirementsfor some embodiments but not for other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, some termsmay be highlighted, for example using italics and/or quotation marks.The use of highlighting has no influence on the scope and meaning of aterm; the scope and meaning of a term is the same, in the same context,whether or not it is highlighted. It will be appreciated that the samething can be said in more than one way. One will recognize that “memory”is one form of a “storage” and that the terms may on occasion be usedinterchangeably.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for some terms are provided. A recital of one or moresynonyms does not exclude the use of other synonyms. The use of examplesanywhere in this specification including examples of any term discussedherein is illustrative only, and is not intended to further limit thescope and meaning of the disclosure or of any exemplified term.Likewise, the disclosure is not limited to various embodiments given inthis specification.

Those skilled in the art will appreciate that the logic illustrated ineach of the flow diagrams discussed above, may be altered in variousways. For example, the order of the logic may be rearranged, substepsmay be performed in parallel, illustrated logic may be omitted; otherlogic may be included, etc.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

We claim:
 1. A computer-implemented method, comprising: retrievingmultiple recordings of conversations associated with multiplerepresentatives, wherein each of the conversations is between at leastone of the representatives and at least one of multiple customers;extracting multiple features from each of the recordings, wherein themultiple features indicate characteristics of any of (a) a customer ofmultiple customers in the corresponding conversation, (b) arepresentative of multiple representatives in the correspondingconversation, (c) the corresponding conversation; analyzing featuresassociated with a first conversation of the multiple conversations todetermine a set of features that is indicative of an action item,wherein the action item is a task to be performed by a first customer ofthe multiple customers or a first representative of the multiplerepresentatives associated with the first conversation; and generating,based on the set of features, information regarding the action item. 2.The computer-implemented method of claim 1, wherein analyzing thefeatures includes determining the set of features based on a usage ofone or more words or phrases in the first conversation that areindicative of the action item.
 3. The computer-implemented method ofclaim 2, wherein determining the set of features includes determining aspecified word or a specified phrase in the first conversation which areexplicitly indicative of the action item.
 4. The computer-implementedmethod of claim 2, wherein determining the set of features includesdetermining a set of words or phrases in the first conversation based onwhich the action item is implicit.
 5. The computer-implemented method ofclaim 4, wherein determining the set of words or phrases in the firstconversation based on which the action item is implicit includesanalyzing a portion of the first conversation prior to or after the setof words or phrases to determine the action item that is implied.
 6. Thecomputer-implemented method of claim 1, wherein generating theinformation regarding the action item includes generating the set offeatures, the set of features including one or more words or phrases inthe first conversation that are indicative of the action item.
 7. Thecomputer-implemented method of claim 1, wherein generating theinformation regarding the action item includes generating a summary ofthe action item.
 8. The computer-implemented method of claim 7, whereingenerating the summary of the action item includes generating thesummary by performing a semantic analysis of the set of features, theset of features including one or more words or phrases in the firstconversation that are indicative of the action item.
 9. Thecomputer-implemented method of claim 7, wherein generating the summaryof the action item includes generating the summary to include a contextof the first conversation, the context including identification of thefirst customer of the first representative.
 10. The computer-implementedmethod of claim 7, wherein the summary includes any information obtainedusing metadata associated with the first conversation.
 11. Thecomputer-implemented method of claim 1, wherein generating the summaryincludes generating the summary using any of a rule-based, an artificialintelligence, a machine learning, or natural language processingtechnique.
 12. The computer-implemented method of claim 1 furthercomprising: determining one or more features that are indicative of anext step, the next step being the action item or any activity otherthan the action item that is expected out of the first customer or thefirst representative subsequent to the first conversation.
 13. Thecomputer-implemented method of claim 12 further comprising: generatinginformation regarding the next step based on the one or more features,the one or more features including one or more words or phrases in thefirst conversation that are indicative of the next step.
 14. Thecomputer-implemented method of claim 1, wherein retrieving the multiplerecordings include retrieving a data stream associated with a real-timeconversation between the first representative and the first customer.15. The computer-implemented method of claim 14 further comprising:extracting one or more features from the real-time conversation that areindicative of the action item.
 16. The computer-implemented method ofclaim 1, wherein extracting the features includes: generating featuresthat include a transcription, vocabulary and a language model of theconversations as a first output.
 17. The computer-implemented method ofclaim 16, wherein extracting the features includes: generating, usingthe first output, features that include semantic information from theconversations.
 18. The computer-implemented method of claim 1, whereinextracting the features includes generating features that include dataregarding conversation flow.
 19. The computer-implemented method ofclaim 1, wherein extracting the features includes generating featuresrelated to a representative-customer pair in a conversation of theconversations.
 20. The computer-implemented method of claim 1, whereinextracting the features includes extracting a visual feature associatedwith a conversation of the conversations.
 21. The computer-implementedmethod of claim 1, wherein extracting the features includes extractingthe features using any of an artificial intelligence, a machinelearning, or natural language processing technique.
 22. Thecomputer-implemented method of claim 1, wherein analyzing the featuresof each of the conversations includes analyzing the features using anyof an artificial intelligence, a machine learning, or natural languageprocessing technique.
 23. The computer-implemented method of claim 1,wherein at least one of the recordings includes a recording of a videocall between one of the customers and one of the representatives. 24.The computer-implemented method of claim 1, wherein at least one of therecordings includes an online meeting between one of the customers andone of the representatives.
 25. The computer-implemented method of claim1, wherein at least one of the recordings includes a recording of avirtual reality-based conversation between one of the customers and oneof the representatives.
 26. The computer-implemented method of claim 1,wherein at least one of the recordings includes a recording of anaugmented reality-based conversation between one of the customers andone of the representatives.
 27. The computer-implemented method of claim1, wherein at least one of the multiple recordings includes an e-mailconversation between one of the customers and one of the multiplerepresentatives.
 28. A non-transitory computer-readable storage mediumstoring computer-readable instructions, comprising: instructions forretrieving multiple recordings of conversations associated with multiplerepresentatives, wherein each of the conversations is between at leastone of the representatives and at least one of multiple customers;instructions for extracting multiple features from each of therecordings, wherein the multiple features indicate characteristics ofany of (a) a customer of multiple customers in the correspondingconversation, (b) a representative of multiple representatives in thecorresponding conversation, (c) the corresponding conversation;instructions for analyzing features associated with a first conversationof the multiple conversations to determine a set of features that isindicative of an action item, wherein the action item is a task to beperformed by a first customer of the multiple customers or a firstrepresentative of the multiple representatives associated with the firstconversation; and instructions for generating, based on the set offeatures, information regarding the action item, the informationincluding (a) one or more words or phrases in the first conversationthat are indicative of the action item or (b) a summary of the actionitem, the summary generated based on the one or more words or phrases inthe first conversation that are indicative of the action item.
 29. Thenon-transitory computer-readable storage medium of claim 28, wherein theinstructions for determining the set of features include: instructionsfor determining a set of words or phrases in the first conversationbased on which the action item is implicit determining the set offeatures.
 30. A system, comprising: a first component that is configuredto extract multiple features from a recording of a conversation, whereinthe conversation is associated with at least one of multiple customersand at least one of multiple representatives; a second component that isconfigured to analyze the features of the conversation to determine aset of features that is indicative of an action item, wherein the actionitem is a task to be performed by a first customer of the multiplecustomers or a first representative of the multiple representatives; anda third component that is configured to generate information regardingthe action item based on the set of features.